System and method for multimedia content protection on cloud infrastructures

ABSTRACT

A system and method for multimedia content protection on elastic cloud infrastructures is presented. The system can be used to protect various multi-media contents, including regular 2D videos, new 3D videos, animated graphics, images, audios clips, songs, and music clips. The system can run on private clouds, public clouds, or any combination of public-private clouds. The system is scalable and cost effective.

DESCRIPTION OF INVENTION

This invention relates to the detection of duplicated content usingcloud systems, and more particularly to a system and method for thedetection of duplicated, copyright material in an online environment.

BACKGROUND

Advances in processing and recording equipment of multimedia contents aswell as the availability of free online hosting sites have made itrelatively easy to illegally duplicate copyrighted materials such asvideos, songs, images, and music clips. Copying and illegallyredistributing multimedia contents over the Internet can result insignificant loss of revenues for content creators. Findingillegally-made copies over the Internet is a complex and computationallyexpensive operation, on account of the huge numbers of availablemultimedia content items across the Internet and the complexity involvedin comparing content items to identify copies.

The present invention seeks to provide a novel system and method formultimedia content protection on cloud infrastructures. The system andmethod can be used to protect various multimedia contents, includingregular 2D videos, new 3D videos, animated graphics, images, audiosclips, songs, and music clips, and can run on private clouds, publicclouds, or any combination of public-private clouds.

Techniques for video copy detection are disclosed in N. Khodabakhshi andM. Hefeeda, Copy detection of 3D videos, In Proc. of ACM MultimediaSystems (MMSys'12) Conference, pp. 131-142, February 2012, Chapel Hill,N.C., USA, J. Bentley, Multidimensional Binary Search Trees used forAssociative Searching, Communications of the ACM, vol 18, No 9, pp.509-517, 1975 and C. Silpa-Anan and R. Hartley, Optimized KD Trees forFast Image Descriptor Matching, In Proc. of IEEE Conference on ComputerVision and Pattern Recognition, pp. 1-8, Anchorage, AL, June 2008

US2010/0318759 discloses a distributed backup storage which supportsdifferential compression. It does not provide a distributed index forperforming fast nearest neighbour searches.

US2011/0064262 discloses attempts to protect 3D videos (videos consistof a base view and an enhancement view) by identifying regions in baseand/or enhancement pictures that are not referenced by other pictures inthe compressed 3D video. Then, these regions are watermarked andtransmitted. Receivers of water-marked videos extract and verify theembedded watermarks.

US2008/0313140 discloses a method and apparatus for multi-dimensionalcontent search and video identification which utilises multidimensionaldatabases and indexes to search different structures such as videos. Todo this, a compact hash of multidimensional vector signatures is used asthe traversal index. For video search applications, global and localsignatures around key points are used as discriminative information ofthe videos. To find matches, a likelihood score based on framesimilarity is computed between the frames of a query video and originalvideos. Then, a correlation between the query video and original videosis generated by using a change in signatures of each sequence of framesin the query video and original videos.

US2008/0178302 discloses the use of video fingerprints to determine 2Dvideo copies.

The present invention provides a method for detecting copies of onlinemultimedia content over distributed systems, the method comprising thesteps of identifying multimedia content to be used as the basis for copydetection, calculating the resources required to extract features fromthe multimedia content, obtaining and deploying the required resources,extracting features from the multimedia content to form signature data,the signature data relating to the extracted features, inputting thesignature data into a distributed index, identifying online content tobe processed for copy detection, calculating the further resourcesrequired to extract features from the online content to be processed,obtaining and deploying the required further resources, extractingfeatures from the online content to form online content data signatures,comparing the signature data with the online content data signatures,and determining whether the online content is a copy of the multimediacontent.

Preferably, the distributed system uses a cloud infrastructure.

Conveniently, the step of extracting features from the multimediacontent is undertaken on a system in the control of the multimediacontent owner.

Advantageously, the signature data includes at least one of a visualsignature, an audio signature, a depth signature, and metadata.

Preferably, the metadata includes details of the content uploader.

Conveniently, the step of forming the signature data includes theformation of a composite signature comprised of a combination of atleast two of: a visual signature, an audio signature, a depth signature,and metadata.

Advantageously, the step of identifying the online content to beprocessed comprises crawling the internet to locate multimedia content.

The present invention also provides a system for detecting copies ofonline multimedia content over distributed systems, the system includingan item or items of multimedia content to be used as the basis for copydetection, a processor to calculate the resources required to extractfeatures from the or each item of multimedia content, a resource, whichwhen deployed, provides a platform on which to extract signature datafrom the multimedia content, a distributed index in which to store thesignature data, online content to be processed for copy detection, afurther resource, which when deployed, provides a platform on which toextract online content data signatures from the online content, and adata comparator, to compare the signature data and the online contentdata signatures, wherein the multimedia content is processed to extractsignature data, the signature data is stored in the distributed index,and the data comparator receives signature data from the distributedindex and compares it against the online content data signatures, todetect online copies of the multimedia content.

Preferably, the resource is a cloud infrastructure.

Conveniently, the further resource is a cloud infrastructure.

Alternatively, the resource used to process the multimedia content is inthe control of the multimedia content owner.

Advantageously, the signature data includes at least one of a visualsignature, an audio signature, a depth signature, and metadata.

Preferably, metadata includes details of the content uploader.

Conveniently, the signature data is formed of a composite signature.

Advantageously, the online content to be processed is identifiedcrawling the internet to locate multimedia content.

Another aspect of the present invention provides a method of creating acomposite signature, the method comprising the steps of calculating avisual signature, based on the visual parts in multimedia objects,calculating an audio signature, based on the audio signal in themultimedia objects, calculating a depth signature, determined based uponthe depth of the multimedia object, collecting metadata, created frominformation associated with multimedia objects, and combining at leasttwo of the visual signature, audio signature, depth signature, andmetadata to form a composite signature.

A yet further aspect of the present invention provides a compositesignature comprising a combination of at least two of: a visualsignature, an audio signature, a depth signature, and metadata.

In order that the present invention may be more readily understood,embodiments of the present invention are described by way of exampleonly, with reference to the accompanying figures, in which:

FIG. 1 shows an overview of the cloud-based content protection systemand method of the present invention;

FIG. 2 shows an overview of the main steps involved in the compositesignature creation of the present invention; and

FIG. 3 shows key processing steps according to the present invention.

The cloud-based multimedia content protection system and method is shownin FIG. 1. The system has multiple components, with a large proportionof the components hosted on cloud infrastructures. In FIG. 1, a generalcase is shown, in which one or more cloud providers can be used by thesystem, because some cloud providers are more efficient and/or providemore cost saving for varying computing tasks. For example, a cloudprovider offering lower cost for inbound bandwidth and storage may beused for downloading and temporarily storing videos from online sites(represented by the upper cloud in FIG. 1), while another cloud provideroffering better compute nodes at lower costs may be used to maintain thedistributed index and to perform the copy detection process (representedby the lower cloud in FIG. 1).

Generally, in the case of content protection systems, there may be threemain parties involved, which may include content owners (the owners ofthe copyrighted materials that need to be protected), hosting sites(online web sites that allow users to upload multimedia contents), andproviders of a content protection service (often third-party companieswhich offer a protection service to content owners).

In an effort to combat the unauthorised copying and distribution ofonline content, the present invention may be deployed and managed by anyof the three parties. Firstly, content owners may deploy a protectionsystem to protect their own content. Further, hosting sites may offer aprotection service by checking their own repositories and reporting tocontent owners. Also, independent third-party companies may offer suchprotection as a service to content owners by periodically checkingcontents posted on online sites.

The present invention may include the following main components, asshown in FIG. 1:

-   -   A distributed index—this may maintain signatures of objects that        need to be protected;    -   RegisterReferenceObjects component—this may create signatures        from the objects that content owners are interested to protect        and may insert them into the distributed index;    -   ProcessQueryObjects component—this may create signatures from        objects downloaded from online sites, which are called query        signatures, and may upload these signatures to a common storage;    -   FindCopies component—this may compare query signatures against        reference signatures in the distributed index to find potential        copies and may also send notifications to content owners if        copies are found; and    -   Crawl component—this may download multimedia content from        various online hosting sites and store the content items in        temporary storage.

The distributed index which maintains signatures of the referenceobjects may be stored and/or hosted on a cloud infrastructure. The indexmay provide fast comparison operations, and various data structures,such as KD trees may be used to implement the index.

The proposed cloud-based index may use varying amount of cloud computingresources on demand. For example, when comparing signatures in largebatches, more machines can be requested from the cloud. Conversely, whenthere are few queries to process, a very few machines can be used tomaintain the index. This leads to substantial saving of recurringoperating costs. Further, the index can be scaled to support massivedata sets.

Also, the index can be deployed quickly, because cloud computingresources can be configured quickly, and can tolerate failures incomputing nodes.

The distributed index may be accessed by an interface, which may includethe following modules:

-   -   Construct—this may build the index from a given set of        signatures;    -   Lookup—if this is given a query signature, it may find the        closest K neighbours to it;    -   mLookup—if this is given a set of signatures, it may find the        closest K neighbours to each of the signatures, and processing        of the signatures may be carried out in parallel on multiple        nodes;    -   Insert—this may insert a new signature into the index;    -   minsert—this may insert a set of signatures in the index in        parallel;    -   Delete—this may delete a signature in the index;    -   mDelete—this may delete a set of signatures in the index in        parallel; and    -   Scale—this may increase or decrease the size of the index to        vary the number of data points held in the index.

The RegisterReferenceObjects and ProcessQueryObjects may share a commonpart, in that they create signatures from multimedia objects.

The RegisterReferenceObjects component may create signatures for objectsthat content owners desire to protect, which may be referred to asreference objects. The ProcessQueryObjects component may createsignatures for objects downloaded from online sites, which may bereferred to as query objects.

The RegisterReferenceObjects component may run as a separateapplication, and may run on servers on content owners' sites. In thiscase, the reference objects do not necessarily need to be transferredoutside the content owners' premises. However, this may require contentowners to deploy a number of servers to run this application. The

RegisterReferenceObjects component may also run on a cloudinfrastructure, which may require the transfer of reference objects tothe cloud and some network costs may apply. In addition, theRegisterReferenceObjects component may insert the generated signaturesinto the distributed index.

The ProcessQueryObjects component may run as a separate application in acloud infrastructure. It may create signatures for query objects storedon the cloud, and may upload them to the storage of query signatures.Query signatures may frequently change and may therefore not be insertedin the distributed indexed. Instead, they may be compared againstreference signatures in the distributed index.

Reference signatures do not, in general, change frequently, because theyare generally based on contractual agreements between the provider ofthe protection service and content owners. However, query objects mayperiodically (often in the order of hours) downloaded from online sites.

Query objects downloaded from online sites are generally stored onlyuntil their signatures have been created by the ProcessQueryObjectscomponent. In case that the content protection system is offered by ahosting site, signatures may be created for the repository of thathosting site.

Both RegisterReferenceObjects and ProcessQueryObjects components maycreate composite (multi-modal) signatures from content objects. Thecontent objects may be any type of multimedia contents, and may include2D videos, 3D videos, images, songs, and music clips. BothRegisterReferenceObjects and ProcessQueryObjects components, in general,require extensive processing of multimedia objects. The presentinvention proposes composite signatures which may include the following:

-   -   A visual signature, which may be created based on the visual        parts in multimedia objects (if they are present) and how the        visual parts may change with time;    -   An audio signature, which may be created based on the audio        signal in the multimedia objects (if they are present) and may        be created using known techniques or a novel method;    -   A depth signature, which, if the or each multimedia objects are        3D, may be created from the perceived (or recorded) depth of the        3D multimedia object;    -   Associated metadata, which may be created from information        associated with multimedia objects such as the file name,        associated tags, the description of the content, the format of        the content, and the IP addresses of their uploader or        downloader or any other suitable attribute; and    -   Any other suitable signature which may be generated from content        data.

FIG. 2 shows the main steps involved in creating a composite signature,and a summary of each step is summarised in the below paragraphs.

The first step of the process is the Split Signals step. This step mayprocess different types of multimedia objects, which may include 2Dvideos, 3D videos, songs, and music clips. Further, it may also be usedto process different media compression methods and file formats. It mayextract separate signals from the multimedia object, and each type ofsignal is handled differently.

The next step of the process is the Create Depth Signature step. If themultimedia object is 3D, then depth signal may be extracted if it isexplicitly encoded with the object. If the depth signal is implicitlyencoded, as is generally the case with stereo video content, a known ornovel depth estimation method may be used, dependent upon the intendedapplication of the present invention. The skilled person will appreciatethat a number of methods exist which may be used.

One such method of creating a depth signature comprises analysing a pairof images, each image containing a plurality of elements, identifying afirst element in one of the pair of images, and identifying pluralelements in the other of the pair of images. The method furthercomprises measuring a disparity parameter between the first element anda set of the plural elements, matching the first element from the set ofplural elements, the matched second element having the smallest measureddisparity parameter, and computing a signature based at least in part onthe measured disparity between the first and second elements.

Then, a Visual Signature may be created. If the multimedia object hasvisual parts, for instance in the case of videos and images, then visualfeatures may be extracted from the video frames of the multimediaobject. Once the visual features have been extracted, any suitablemethod may be employed to create the Visual Signature, for instance themethod detailed in Khodabakhshi and Hefeeda. The skilled person willappreciate that any suitable method may be employed.

Next, an Audio Signature may be created. If the multimedia object hasaudio parts, as is the case with video with audio and songs, variousmethods for creating audio signatures may be used, again dependent uponthe intended use.

Once the above signatures have been created, a Combined Signature may becreated. This step may combine different signatures and may assigndifferent weights to each of them. It may also analyse metadataassociated with the multimedia object and may extract importantinformation that can be used in the copy detection process. Thisimportant information may include (but is not limited to) the format ofthe object, the type of content, the number of downloads of the object,the IP address of the uploader, and any other suitable information.

It is to be understood that, with the exception of creating the CombinedSignature, the above steps may be carried out in any suitable order, aswill be understood by the skilled reader.

All of the above steps may be performed for each multimedia object. TheRegisterReferenceObjects and ProcessQueryObjects components should beable to handle many objects at the same time, and therefore the presentinvention presents a distributed design for these components. Thepresent invention uses cloud infrastructure in an elastic fashion,increasing and decreasing the resources used dependent upon the size ofthe system and the intended use. The elastic use of the cloud is allowsfor cost efficiency and fast processing.

The distributed design for the present invention is set out in FIG. 3,and may include features as discussed below.

The distributed design may obtain information about the number ofobjects that need to be processed, which is the workload size, then mayestimate the required resources to process the whole workload, within aspecified completion time. When the required resources have beenestimated, the process may then request computing resources from thecloud infrastructure.

Based on the returned computing resources and their configurations(e.g., number of nodes, and number of processor cores and memory size oneach node), the process may then partition the workload and may allocateit to different computing nodes. On-line monitoring of the progress ofeach node may be performed and task reallocation may be undertaken ifsome nodes fail or lag behind. Each computing node may compute thecomposite signatures of the given multimedia objects using the steps setout above and shown in FIG. 2, and may then write the resultingsignature on a common cloud storage.

The present invention presents a FindCopies component in addition to theabove, which may compare the query signatures computed by theProcessQueryObjects component versus the signatures stored in thedistributed index, which may be created by the RegisterReferenceObjectscomponent. The FindCopies component may use the interface provided bythe distributed index.

Similar to the ProcessQueryObjects and RegisterReferenceObjectscomponents, the FindCopies component may be computationally expensive,especially if the distributed index is large and there are many querysignatures.

Therefore, the FindCopies component may run on elastic cloudinfrastructures. This design is similar to the distributed design inFIG. 3, and it may first estimate the workload size based on the numberof the query signatures and the current size of the distributed index.It may then request enough cloud computing resources to process thisworkload, then may allocate partitions of the query signatures todifferent computing nodes. Finally, if any copies are found, thecorresponding content owners may be notified.

If the content protection system is being run by online hosting sites,then the Crawl component is not needed, since the objects are availableon the hosting sites' archives. The Crawl component is needed if thesystem is run by the content owners or third-part companies providingthe content protection service.

The number of multimedia objects added to hosting sites on a daily basisis huge. Therefore, the present invention may crawl the internet to findmultimedia objects to be checked for copies, by way of a Crawlcomponent.

Given that the potential online information to be crawled is very large,the Crawl component is designed efficiently. The Crawl component may runon cloud infrastructures. Similar to the previous three components, itmay first request cloud resources to run multiple instances of the Crawlcomponent. The different instances may download different multimediaobjects, or alternatively, may all seek the same multimedia objects. Thedifferent instances of the Crawl component may be managed by acoordinator that may allocate download tasks to the different Crawlinstances. To do so, the coordinator may contact various hosting sitesand collects addresses and metadata associated with available multimediaobjects.

Then, the coordinator may construct various queries and repeatedlycontacts hosting sites to construct a master file containing uniqueobjects that need to be downloaded. The queries constructed by thecoordinator may be used to limit the scope of the objects that will bedownloaded. For example, queries submitted by the coordinator mayrequest objects that have been posted within a given time period, havecertain lengths, have been watched more than given thresholds, uploadedor watched from certain geographic areas, or any other suitablecriteria.

Once the master file is constructed, partitions of this file may beallocated to different Crawl instances. Then, each instance may startdownloading objects allocated to it and may store them in cloud storage.

In use, a particular content object or collection of content objects tobe used as the basis for copy detection would be identified, andsignatures created which are unique to the or each content object, usingthe steps set out above. The cloud resources required would beestimated, and the required cloud resources would be procured, as setout in the discussions of the RegisterReferenceObjects andProcessQueryObjects sections above.

Then, the signatures would be stored in common cloud storage, and thedata regarding the content objects and the signatures, would be storedin a distributed index. If the copy detection is being run remotely(i.e. not on a hosted content provider's systems), the web (or othernetwork) would be need to be crawled to find hosted content objects,using the Crawl method as detailed above. The located hosted contentobjects would then be analysed using FindCopies (again, as discussedabove) to extract the signatures from the hosted content objects, usingfurther cloud resources.

Then, the hosted content objects which have the same attributes (andsignatures) as the identified content objects would be identified andflagged as potential copies, again using FindCopies.

When used in this specification and claims, the terms “comprises” and“comprising” and variations thereof mean that the specified features,steps or integers are included. The terms are not to be interpreted toexclude the presence of other features, steps or components.

The features disclosed in the foregoing description, or the followingclaims, or the accompanying drawings, expressed in their specific formsor in terms of a means for performing the disclosed function, or amethod or process for attaining the disclosed result, as appropriate,may, separately, or in any combination of such features, be utilised forrealising the invention in diverse forms thereof.

1. A method for detecting copies of online multimedia content overdistributed systems, the method comprising the steps of: identifyingmultimedia content to be used as the basis for copy detection;calculating the resources required to extract features from themultimedia content; obtaining and deploying the required resources;extracting features from the multimedia content to form signature data,the signature data relating to the extracted features; inputting thesignature data into a distributed index; identifying online content tobe processed for copy detection; calculating the further resourcesrequired to extract features from the online content to be processed;obtaining and deploying the required further resources; extractingfeatures from the online content to form online content data signatures;comparing the signature data with the online content data signatures;and determining whether the online content is a copy of the multimediacontent.
 2. The method of claim 1, wherein the distributed system uses acloud infrastructure.
 3. The method of claim 1, wherein the step ofextracting features from the multimedia content is undertaken on asystem in the control of the multimedia content owner.
 4. The method ofclaim 1, wherein the signature data includes at least one of a visualsignature, an audio signature, a depth signature, and metadata.
 5. Themethod of claim 4 wherein the metadata includes details of the contentuploader.
 6. The method of claim 4 wherein the step of forming thesignature data includes the formation of a composite signature comprisedof a combination of at least two of: a visual signature, an audiosignature, a depth signature, and metadata.
 7. The method of claim 1wherein the step of identifying the online content to be processedcomprises crawling the internet to locate multimedia content.
 8. Asystem for detecting copies of online multimedia content overdistributed systems, the system including: an item or items ofmultimedia content to be used as the basis for copy detection; aprocessor to calculate the resources required to extract features fromthe or each item of multimedia content; a resource, which when deployed,provides a platform on which to extract signature data from themultimedia content; a distributed index in which to store the signaturedata; online content to be processed for copy detection; a furtherresource, which when deployed, provides a platform on which to extractonline content data signatures from the online content; and a datacomparator, to compare the signature data and the online content datasignatures, wherein the multimedia content is processed to extractsignature data, the signature data is stored in the distributed index,and the data comparator receives signature data from the distributedindex and compares it against the online content data signatures, todetect online copies of the multimedia content.
 9. The system of claim8, wherein the resource is a cloud infrastructure.
 10. The system ofclaim 8, wherein the further resource is a cloud infrastructure.
 11. Thesystem of claim 8 wherein the resource used to process the multimediacontent is in the control of the multimedia content owner.
 12. Thesystem of claim 8 wherein the signature data includes at least one of avisual signature, an audio signature, a depth signature, and metadata.13. The system of claim 12 wherein the metadata includes details of thecontent uploader.
 14. The system of claim 12 wherein the signature datais formed of a composite signature.
 15. The system of claim 8 whereinthe online content to be processed is identified crawling the internetto locate multimedia content.
 16. A method of creating a compositesignature, the method comprising the steps of: calculating a visualsignature, based on the visual parts in multimedia objects; calculatingan audio signature, based on the audio signal in the multimedia objects;calculating a depth signature, determined based upon the depth of themultimedia object; collecting metadata, created from informationassociated with multimedia objects; and combining at least two of thevisual signature, audio signature, depth signature, and metadata to forma composite signature.
 17. (canceled)